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(54) Image processing apparatus for a robot 

(57) A robot control unit 42 and an image processing 
control unit (control unit of the image processing appa- 
ratus) 43 are incorporated into a robot controller 40. A 
camera CM is connected to the image processing con- 
trol unit 43. A main body 1 of a robot is connected to the 
robot control unit 42 through an amplifier 41 . A portable 
robot teaching pendant 80 connected to the robot con- 
trol unit 42 is provided with a monitor display, and func- 
tions also as a teaching pendant of the image process- 
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ing apparatus. Therefore, by using the teaching pendant 
80, manipulation of image processing, and issuing of an 
instruction to a program for processing an image can be 
performed. Furthermore, an image obtained by a cam- 
era CM, and information relevant to the manipulation of 
the image processing apparatus such as an operation 
menu, etc. can be displayed on the monitor display. 
Therefore, an operator can efficiently perform all oper- 
ations relevant to the robot, the camera, processing an 
image, etc. while watching a monitor screen. 
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Description 

[0001] The present invention relates to an image 
processing apparatus for a robot for locating and check- 
ing an object by functioning as a visual unit of a robot, 5 
and more specifically to a technology for improving the 
manipulation of the image processing apparatus tor a 
robot. 

[0002] A robot is used in combination with an image 
processing apparatus provided with a camera for taking to 
a photograph of a workpiece, etc. in many cases. The 
image processing apparatus used in combination with 
a robot (hereinafter referred to as an image processing 
apparatus for a robot) can be embodied as described 
below in (1) through (4). 15 

(1 ) As shown in Fig. 1 , the image processing appa- 
ratus (main body section) is connected to a robot 
controller for controlling a robot (main body), a TV 
monitor, a camera, and a teaching device (a teach- 20 
ing pendant) for image processing apparatus. A 
control unit for manipulating image processing is in- 
dependent of the control unit for controlling a robot, 
and both are connected through a communications 
unit. 25 

The teaching pendant for the image processing 
apparatus is provided independent of a portable 
teaching pendant (robot teaching pendant) con- 
nected to the robot controller, and is used for ma- 
nipulating the image processing apparatus and 30 
teaching a program for processing an image, etc. 
The information about the image obtained by the 
camera, the processed image, the information on 
the manipulation of the image processing appara- 
tus such as an operation menu, etc. are displayed 35 
on a TV monitor independent of the robot teaching 
pendant and the teaching pendant for the image 
processing apparatus. 

(2) As shown in Fig. 2, a desktop-type personal 
computer is used as a main body section of the im- 40 
age processing apparatus. The personal computer 
used as the image processing apparatus is con- 
nected to the robot controller for controlling a robot 
(main body) and the camera. The control unit of the 
personal computer is provided as independent of 45 
the control unit of the robot controller, and these 
control units are connected to each other through 
the communications unit. 

A keyboard and a mouse provided for the per- 
sonal computer are used as a teaching pendant for so 
the image processing apparatus, and manipulation 
for processing an image and teaching of a program 
are carried out through this teaching pendant. The 
information about the image obtained by the cam- 
era, the processed image, the information on ma- ss 
nipulation of the image processing apparatus such 
as an operation menu, etc. are displayed on the 
monitor display (CRT, LCD. etc.) provided for the 



personal computer. 

(3) As shown in Fig. 3, the robot controller and the 
main body section of the image processing appara- 
tus are used as a built-in apparatus. The TV moni- 
tor, the camera, the teaching pendant for the image 
processing apparatus, and the robot teaching pen- 
dant are connected to the built-in apparatus. The 
control unit for controlling a robot and the control 
unit for manipulating image processing are shared 
(connected through a bus). 

The teaching pendant for the image processing 
apparatus is provided independent of the portable 
robot teaching pendant, and operations for 
processing an image or the like and teaching a pro- 
gram for processing an image are carried out 
through this teaching pendant. The information 
about the image obtained by the camera, the proc- 
essed image, the information on manipulation of the 
image processing apparatus such as an operation 
menu, etc. are displayed on the TV monitor inde- 
pendent of the robot teaching pendant or the teach- 
ing pendant for the image processing apparatus. 

(4) Described above are typical embodiments of the 
image processing apparatus for a robot. In addition, 
as disclosed in Japanese Patent Application Laid- 
Open NO. 5-66825, a TV monitor is detachably 
mounted on the teaching pendant for the image 
processing apparatus. The teaching pendant for the 
image processing apparatus is provided independ- 
ent of the robot teaching pendant. 

[0003] Incidentally, manipulation of the image 
processing apparatus for a robot is often carried out 
while frequently referring to image information. There- 
fore, it is desired that an operation menu and the image 
information are displayed near the operator. Otherwise, 
the operator has to frequently switch the sight, thereby 
lowering the operability. 

[0004] Furthermore, since the image processing ap- 
paratus is used for a robot, target workpiece and a cam- 
era are positioned beside the robot. As a result, it is often 
necessary to conduct a detection test by shifting the po- 
sition of target workpiece in the field of the vision of the 
camera, or to adjust the focalization and the stop of the 
lens of the camera. Accordingly, to efficiently perform 
these operations, it is desired that means for manipulat- 
ing an image processing apparatus is of a portable type. 
[0005] Furthermore, since the image processing ap- 
paratus is used for a robot, the manipulation of the robot 
and the manipulation of the image processing apparatus 
are alternately performed in most cases. Therefore, it is 
desired that the robot teaching pendant can also manip- 
ulate the image processing apparatus. Otherwise, an 
operator has to hold one teaching pendant at one time 
and the other teaching pendant at the other time or has 
to move from one place to the other place each time the 
manipulation is switched from the robot to the image 
processing apparatus and vice versa. 
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[0006] Considering these demerits, the conventional 
technology described above in (1) through (4) has a 
problem in easiness in manipulation. That is, the oper- 
ator who carries a portable robot teaching pendant and 
is near a robot (main body) has the difficulty in perform- s 
ing all manipulations of the image processing appara- 
tus. It is desired that, by transforming the above de- 
scribed conventional technology described above in (3), 
the manipulations relevant to image processing can be 
performed through a robot teaching pendant by, for ex- 10 
ample, using a robot teaching pendant also as a teach- 
ing pendant for the image processing apparatus. In this 
case, a detachable TV monitor is required, and the prob- 
lem that additional handling and cost are required re- 
mains. 75 
[0007] Under the above described current situation, 
the present invention aims at solving the problems of 
the conventional technology by improving manipulation 
of the image processing apparatus for a robot. 
[0008] To attain the above described purpose, the so 
present invention provides an image processing appa- 
ratus manipulating menu, and a display unit for display- 
ing monitor image information, on a portable teaching 
pendant for the image processing apparatus for per- 
forming the manipulations relevant to processing an im- 25 
age. 

[0009] According to a first embodiment of the image 
processing apparatus for a robot of the present inven- 
tion, the image processing apparatus is built in a robot 
controller, and has a portable teaching pendant con- 30 
nected thereto. 

[0010] The image processing apparatus includes: a 
unit for fetching an image from a camera; memory for 
storing image data from the camera or intermediate im- 
age data obtained in a stage of image processing; and 35 
a unit for converting image data from the camera, the 
image data from the camera stored in the memory, or 
the intermediate image data into a gray scale or a color 
scale. 

[0011] Then, the portable teaching pendant includes 40 
a display unit and a unit for manipulating image process- 
ing. The display unit displays an image and indication 
for manipulation of image processing, simultaneously, 
or in a switching mode or superposition mode. 
[0012] According to a second embodiment of the im- 45 
age processing apparatus for a robot of the present in- 
vention, the image processing apparatus is designed in- 
dependent of the robot teaching pendant, and has a 
portable teaching pendant connected thereto. 
[0013] The image processing apparatus includes: a so 
unit for fetching an image from a camera; memory for 
storing image data from the camera or intermediate im- 
age data obtained in a stage of image processing; and 
a unit for converting image data from the camera, the 
image data from the camera stored in the memory, or ss 
the intermediate image data into a gray scale or a color 
scale. 

[0014] Then, the portable teaching pendant includes 



a display unit and a unit for manipulating image process- 
ing. The display unit displays an image and indication 
for manipulation of image processing, simultaneously, 
or in a switching mode or superposition mode. 
[0015] Preferably, the above described first and sec- 
ond embodiments include a unit for specifying an image 
processing for an image displayed on the display unit 
by overlapping geometric graphics with the image and 
displaying them together according to the operation pro- 
cedure of the above described image processing. 
[0016] According to a third embodiment of the image 
processing apparatus for a robot of the present inven- 
tion, the image processing apparatus is built in a robot 
controller, and includes: a unit for fetching an image from 
a camera; memory for storing image data from the cam- 
era or intermediate image data obtained in a stage of 
image processing; and a unit for converting image data 
from the camera, the image data from the camera stored 
in the memory, or the intermediate image data into a 
gray scale or a color scale. The robot controller has a 
portable teaching pendant connected thereto through a 
cable. Furthermore, the teaching pendant includes: a 
unit for generating or editing a program of the robot; a 
unit for manipulating the robot: and a display unit. This 
teaching pendant can display on the display unit an im- 
age converted into the gray scale. 
[001 7] Furthermore, the teaching pendant includes a 
unit for manipulating image processing. The display unit a* 
displays indication for use in generating or editing a pro- % 
gram of a robot and indication for manipulation of image *:>. 
processing, in addition to an image, simultaneously, or 
in a switching mode or superposition mode. 
[0018] Preferably, the third embodiment according to 
the present invention includes a unit for specifying an 
image processing for an image displayed on the display 
unit by overlapping geometric graphics with the image 
and displaying them together by means of a manipula- 
tion unit for the image processing. 
[0019] In the first through the third embodiments of 
the present invention, a part of the manipulation unit of 
the teaching pendant can be designed as a touch panel. 
In addition, it is desired that the apparatus further in- 
cludes a unit for incorporating an instruction for image 
processing into a robot program. 
[0020] The foregoing and other objects and features 
of the invention will become apparent from the following 
description of preferred embodiments of the invention 
with reference to the accompanying drawings, in which: 

Fig. 1 is a schematic diagram of a first example of 
the image processing apparatus for a robot accord- 
ing to the conventional technology; 
Fig. 2 is a schematic diagram of a second example 
of the image processing apparatus for a robot ac- 
cording to the conventional technology; 
Fig. 3 is a schematic diagram of a third example of 
the image processing apparatus for a robot accord- 
ing to the conventional technology; 
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Fig. 4 is a schematic diagram of a first embodiment 
of the image processing apparatus for a robot ac- 
cording to the present invention; 
Fig. 5 is a schematic diagram of a second embodi- 
ment of the image processing apparatus for a robot 
according to the present invention; 
Fig. 6 is a schematic diagram of a third embodiment 
of the image processing apparatus for a robot ac- 
cording to the present invention; 
Fig. 7 is a block diagram of the principal part of the 
image processing control apparatus shown in Figs. 
4 to 6; 

Fig. 8 is a block diagram of the principal part of the 
teaching pendant connected to the image process- 
ing apparatus shown in Figs. 4 and 5; 
Fig. 9 is a block diagram of the principal part of the 
image processing apparatus shown in Fig. 6; 
Fig. 10 is a schematic diagram of the front view of 
the first example of the teaching pendant connected 
to the image processing control apparatus shown 
in Figs. 4 to 6; 

Fig. 11 is a schematic diagram of the front view of 
the second example of the teaching pendant con- 
nected to the image processing control apparatus 
shown in Figs. 4 to 6; 

Fig. 12 shows an example of the contents displayed 
on the monitor screen of the teaching pendant 
shown in Fig. 10 or 11; 

Fig. 1 3 is a flowchart of the setup operation for start- 
ing actual work for the image processing control ap- 
paratus shown in Figs. 4 to 6; and 
Fig. 14 is a flowchart of the operation of actual work 
of the image processing control apparatus shown 
in Figs. 4 to 6. 

[0021] The first, second, and third embodiments of the 
image processing apparatus for a robot according to the 
present invention will be described below by referring to 
the schematic diagrams of Fig. 4 to Fig. 6. 
[0022] According to the first embodiment shown in 
Fig. 4, a robot control unit 12 and an image processing 
control unit (control unit of the image processing appa- 
ratus) 13 are incorporated into a robot controller 10. A 
main body 1 of a robot is connected to a robot control 
unit 12 through an amplifier 11 

[0023] A portable teaching pendant (teaching console 
panel) 60 provided with a camera CM and a monitor dis- 
play is connected to the image processing control unit 
13. The teaching pendant 60 is provided independent 
of a robot teaching pendant (robot teaching console 
panel) 50 connected to the robot control unit 12. The 
teaching pendant 60 is used for manipulating image 
processing, teaching of an image processing program, 
etc. In addition, an image obtained by a camera, a proc- 
essed image, and information relevant to the manipula- 
tions of the image processing apparatus such as an op- 
eration menu are displayed on the monitor display pro- 
vided on the teaching pendant 60. 



[0024] On the other hand, according to the second 
embodiment shown in Fig. 5, a robot controller 20 is de- 
signed independent of an image processing control unit 
(control unit of the image processing apparatus) 30. The 

s main body 1 of the robot is connected to a robot control 
unit 22 through an amplifier 21 in the robot controller 20. 
[0025] A portable teaching pendant (teaching console 
panel) 70 provided with a camera CM and a monitor dis- 
play is connected to the image processing control unit 

10 30. The teaching pendant 70 is provided independent 
of a robot teaching pendant 50 to be connected to the 
robot control unit 22. The teaching pendant 70 is used 
for manipulation for image processing, teaching of an 
image processing program, etc. In addition, an image 

15 obtained by a camera, a processed image, and informa- 
tion relevant to the manipulation of the image process- 
ing apparatus such as an operation menu, etc. is dis- 
played on the monitor display provided on the teaching 
pendant 70. Furthermore, according to the third embod- 

20 iment shown in Fig. 6, a robot control unit 42 and an 
image processing control unit (control unit of the image 
processing apparatus) 43 are incorporated into a robot 
controller 40. A camera CM is connected to the image 
processing control unit 43. The main body 1 of the robot 

25 is connected to the robot control unit 42 through an am- 
plifier 41. 

[0026] A portable teaching pendant (teaching console 
panel) 80 provided with a monitor display is connected 
to the robot control unit 42. The teaching pendant 80 
30 serves both as a robot teaching pendant and a teaching 
pendant for the image processing apparatus. 
[0027] That is, the teaching pendant 80 corresponds 
to a combination of the portable robot teaching pendant 
50 with the teaching pendant 70 shown in Fig. 5 (second 
35 embodiment), and is used for manipulation of image 
processing and teaching of an image processing pro- 
gram, in addition to operation for a normal teaching to 
a robot. Furthermore, an image obtained by a camera, 
a processed image, and information relevant to manip- 
40 ulation of the image processing apparatus such as an 
operation menu, etc. can be displayed on the monitor 
display provided on the teaching pendant 80. 
[0028] Next, the image processing control units 13, 
30, and 43 according to the above described embodi- 
es ments (Figs. 4 through 6) will be described below by re- 
ferring to the block diagram of the principal part shown 
in Fig. 7. 

[0029] Each of the image processing control units 1 3, 
30, and 43 comprises a central processing unit (CPU) 
so to which a camera interface, a console interface, a mon- 
itor interface, a communications interface, image mem- 
ory, program memory, and data memory are connected 
through a bus. 

[0030] A camera for taking a photograph of an object 
55 sequentially transmitted to, for example, the production 
line, is connected to the camera interface, and an image 
of the object processed as a gray scale image is stored 
in the image memory. The CPU processes the image 



4 



EP 0 997 236 A1 



8 



stored in the memory, identifies an object, and detects 
the position and the posture of the object. 
[0031] A portable teaching pendant of the image 
processing apparatus is connected to the console inter- 
face. A teaching pendant and a monitor display (for ex- 
ample, a TV monitor, an LCD) are connectedto the mon- 
itor interface so that an image stored in the image mem- 
ory, a picture captured by a camera, etc. can be dis- 
played on them. 

[0032] A robot control unit is connected to the com- 
munications interface. The control program which the 
CPU uses in controlling the image processing apparatus 
is stored in the program memory. An image analysis al- 
gorithm is also contained in the control program. Data 
memory is made of a non-volatile RAM storing various 
data such as various setting parameter, etc. of a user- 
generated application program and image analyzing 
method. 

[0033] A first feature of the image processing appara- 
tus for a robot according to the present invention is to 
use a portable teaching pendant as a console of the im- 
age processing apparatus, and display image informa- 
tion outputted from the monitor interface on the monitor 
screen provided there, which is described later. The im- 
age output from the monitor interface is also connected 
to the TV monitor, in addition to the teaching pendant. 
This is to display a result of an image processing, etc. 
on a relatively large monitor screen for a large number 
of users during operation at a manufacturing line. Obvi- 
ously, the TV monitor is not an essential component, so 
that the result of an image processing an image may be 
displayed on a monitor display provided on the teaching 
pendant during operation at a production line. 
[0034] Furthermore, a second feature of the image 
processing apparatus for a robot according to the 
present invention is, as understood from in Fig. 4 (first 
embodiment), that the control unit of the image process- 
ing apparatus does not have any exclusive image proc- 
essor. This is because a gray scale image processing 
can be carried out by the CPU, without using any exclu- 
sive processor, thanks to progress in hardware in recent 
years. Of course, it is obvious that an exclusive image 
processor can be provided to process an image. 
[0035] The configuration of the teaching pendant 60 
connected to the image processing control unit 13 
shown in Fig. 4 (first embodiment) will be described be- 
low by referring to Fig. 8. 

[0036] In Fig. 8, the teaching pendant 60 is connected 
to the image processing control unit 1 3 in the robot con- 
troller 10. The image processing control unit 13 is, as 
described above, incorporated into the robot controller 
10, and is built in the robot controller 10 with the robot 
control unit 12 for controlling the main body 1 of the ro- 
bot. 

[0037] In the teaching pendant 60, a processor, flash 
memory, a DRAM (direct memory access controller), a 
communications port, a PCMCIA (Personal Computer 
Memory Card International Association) interface, an 



AD (analog/digital) converter to which a touch panel is 
connected, a graphic controller to which a liquid crystal 
display (LCD) is connected : and a key control circuit to 
which a cursor key, the numerical keys, etc. for use in 
5 processing an image are connected through a bus. 
[0038] The flash memory stores an operating system 
program for the processor. For example, an operating 
system program (for example, 'Windows CE\ of Micro- 
soft, etc.) for supporting a graphic user interface is 
10 stored. This program converts a program, data on pa- 
rameters, etc., relevant to processing an image, into the 
expression format of visual diagrams and graphics so 
that they can be displayed on the liquid crystal display 
as a display unit. 
is [0039] The DRAM is used as work memory such as a 
temporary storage, etc. of data for use in performing 
arithmetic operations. The communications port con- 
nects, as described above, the image processing con- 
trol unit 1 3 or 43 with the teaching pendant 60 or 80 
20 through a cable. 

[0040] The PCMCIA interface reads a program rele- 
vant to an image processing stored in the memory card 
and data on various parameters and writes data to be 
stored in the DRAM in the memory card. 
2$ [0041] The touch panel is arranged on the liquid crys- 
tal display in piles, so that touching an area for one of 
the items displayed on the liquid crystal display will en- 
able a command for the item to be inputted. Such an •** 
inputting method has been well-known. u 
30 [0042] The key control circuit comprises a processor £ 
or a DMAC (Direct Memory Access Controller), and 
processes signals from various key, etc. In addition, the 
processor may directly process a key. 
[0043] The configuration of the teaching pendant 70 
3S connected to the image processing control unit 30 
shown in Fig. 5 (second embodiment) is the same as 
the configuration of the teaching pendant 60 shown in 
Fig. 8. However, although the teaching pendant 70 
shown in Fig. 8 (first embodiment) is connected to the 
40 image processing control unit 13 built in the robot con- 
troller, the teaching pendant 70 shown in Fig. 5 is con- 
nected to the image processing control unit 30 provided 
externally to the robot controller 20 as described above. 
[0044] The configurations of the robot control unit 42 
45 shown in Fig. 6 (third embodiment), and the teaching 
pendant 80 to be connected to the robot control unit 42 
will be described below by referring to Fig. 9. 
[0045] in Fig. 9, the teaching pendant 80 is connected 
to the robot control unit 42 of the robot controller 40 for 
50 controlling the main body 1 of the robot through a cable 
and a signal line. The teaching pendant 80 is a portable 
teaching pendant provided with a monitor display as de- 
scribed above, and serves both functions as a robot 
teaching pendant and a teaching pendant for the image 
ss processing apparatus. The teaching pendant 80 is con- 
nected to the robot control unit 42 through a cable and 
. a signal line. 
[0046] The robot control unit 42 comprises a proces- 
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sor, ROM for storing a system program, etc. for control- 
ling the processor, RAM used to store an instructing op- 
eration program and various parameters, etc., and used 
to temporarily store data in performing arithmetic oper- 
ations, etc., a servo-control circuit for drive controlling 
the servo-motor of each axis ot the main body 1 of the 
robot, a communications controller for controlling the 
communications with the teaching pendant BO, a PCM- 
CIA interface, each connected to processor through a 
bus. 

[0047] The servo-amplifier 41 of each axis is connect- 
ed to the servo -control circuit. A servo motor of each 
axis of the main body 1 of the robot is connected to the 
servo-amplifier. In addition, the servo-amplifier 41 is 
connected with an emergency stop circuit so that the 
operation of the robot can be stopped by turning off the 
power supply to the servo-amplifier when an emergency 
stop signal is transmitted from a manual console panel. 
[0048] In addition, the PCMCIA interface is used tor 
reading information from a memory card, and the proc- 
essor reads information on an operation program, etc. 
of a robot stored in the memory card through the PCM- 
CIA interface. The read information is stored in RAM. 
[0049] The teaching pendant 80 and the image 
processing control unit 43 (refer to Fig. 6) are connected 
to the communication controller through a cable. The ro- 
bot control unit receives an instruction and data from the 
teaching pendant 80 and the image processing control 
unit 43, and transmits robot controlling data, image 
processing data, etc. to the teaching pendant 80. 
[0050] Then, the processor reads robot controlling da- 
ta stored in the RAM, and, if move command is issued, 
then the processor outputs the move command to the 
servo-control, circuit based on the instructing operation 
program. The servo-control circuit carries out feed back 
control of position, speed and electric current to control 
the servo-amplifier 41 to drive the servo-motor of each 
axis of the main body 1 of the robot, thereby operating 
the robot. 

[0051] In the teaching pendant 80, a processor, flash 
memory, a DRAM, a communications port, a PCMCIA 
interface, an AD (analog/digital) converter to which a 
touch panel is connected, a graphic controller to which 
a liquid crystal display (LCD) is connected, and a key 
control circuit to which a jog key, an emergency stop key, 
a cursor key, numerical keys, etc. for use in processing 
an image are connected with one another through a bus. 
[0052] The flash memory stores an operating system 
program for the processor. For example, an operating 
system program (for example, 'Windows CE\ etc. of Mi- 
crosoft, etc.) for supporting a graphic user interface 
which can be configured in, for example, a semiconduc- 
tor memory is stored. This program converts a robot op- 
eration program, an image processing program, data on 
parameters, etc. into the expression format of visual di- 
agrams and graphics so that they can be displayed on 
the liquid crystal display as a display unit. 
[0053] The DRAM is used as work memory for a tem- 



porary storage, etc. of data for use in performing arith- 
metic operations. The teaching pendant 80 may also be 
designed such that it can generate control data for a ro- 
bot and data relevant to an image processing, read an 

s operation program and parameter stored in the RAM of 
the robot control unit, store the control data and param- 
eter, and modify the control data. 
[0054] The communications port connects the robot 
control unit 42 with the teaching pendant 60, and is con- 

10 nectedtothe communications controller through a cable 
for serial communications. 

[0055] The PCMCIA interface reads a robot operation 
program, a program relevant to an image processing 
and data on various parameters stored in the memory 

is card, and writes operation program and various param- 
eters stored in the DRAM, into the memory card. 
[0056] The touch panel is arranged on the liquid crys- 
tal display in piles, so that touching an area for one of 
the items displayed on the liquid crystal display will en- 

20 able a command for the item to be inputted. This input- 
ting operation using such a touch panel is conventional 
one. 

[0057] The key control circuit comprises a processor 
and a DM AC, and processes a signal from a jog key, an 

25 emergency stop key, a cursor key for processing an im- 
age, numerical keys (refer to Fig. 7), etc. The processor 
may directly conduct processing of the keys. 
[0058] An example of the teaching pendants 60 and 
70 to be connected to the image processing control units 

30 13 and 30, which can be used in the case of Fig. 4 (first 
embodiment) and Fig. 2 (second embodiment) will be 
described below by referring to Fig. 10. Fig. 10 shows 
a front view of the teaching pendants 60 and 70. 
[0059] The teaching pendants 60 and 70 are provided 

35 with a cursor key CK, numerical keys TK, and a monitor 
display (TV or LCD) M as shown in Fig. 1 0, and performs 
manipulations relevant to image processing, displays 
image information, etc. 

[0060] Furthermore, an example of the teaching pen- 

40 dant 80 to be connected to the robot control unit 42, 
which can be used in the case of Fig. 6 (third embodi- 
ment) will be described below by referring to Fig. 11 . Fig. 
11 shows a front view of the teaching pendant 80. 
[0061] The teaching pendant 80 is provided with not 

45 only a cursor key CK, numerical keys TK, a monitor dis- 
play (TV or LCD)M but also an emergency stop button 
EB, a manual feed key JK, a deadman's switch DS, etc. 
as exclusive keys. This teaching pendant 80 serves both 
as a robot teaching pendant and a teaching pendant for 

so the image processing apparatus (for operations relevant 
to image processing, displaying image information, etc.) 
The deadman's switch DS is used to u n cond it ion ally 
stop a robot when an operator stops pressing this 
switch, guaranteeing the safety of the operator when 

55 driving a robot. 

[0062] On the monitor display screen M, a robot op- 
eration menu, an image processing apparatus manipu- 
lating menu, image information, etc. can be displayed 
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by operating the cursor key CK, the numerical keys 10, 
etc. Fig. 12 shows the state of displaying the image on 
the left of the monitor screen M and the operation menu 
of the image processing apparatus on the right. Switch- 
ing to or from the robot operation menu is performed s 
using a soft key provided on an outer frame (not shown) 
of the display unit. A touch panel is provided on the sur- 
face of the display unit of the teaching pendant, and the 
input information from the touch panel is transmitted to 
the console interface of a robot or the console interface io 
of the image processing apparatus depending on the 
contents displayed on the touch panel. 
[0063] The setup operation for starting work to be per- 
formed by using the image processing apparatus for a 
robot according to the present invention (first through is 
third embodiments) will be described below by referring 
to the flowchart shown in Fig. 13. The above-mentioned 
work (application to be performed) involves recognizing 
the shift of the position of a target workpiece using a 
camera CM, adjusting and controlling the robot opera- 20 
tion program, and performing the program. 
[0064] First, a camera CM is set at a predetermined 
position of a cell (work area), and connected to a camera 
interface through a camera cable (step S1 1 ). The cam- 
era cable contains a picture signal line and a synchro- 25 
nous signal line. The power supply for driving the cam- 
era can be externally provided, or from the image 
processing apparatus by providing a power source line 
for the camera cable. 

[0065] An item for display of a raw image from the 30 
camera is selected on the image processing apparatus 
menu of the teaching pendant. Then, a raw image of the 
gray scale of a camera is displayed on the teaching pen- . 
dant. It is desired that a raw image is updated 30 times 
per second. However, the frequency of several times per 35 
second is acceptable as the minimal update frequency. 
It is also desired that the gray scale level of a displayed 
image is as high as the level of the image memory. For 
example, assuming that the image memory has 256 lev- 
els of a gray scale, the gray scale resolution can be low- 40 
ered to 16 levels when an image is displayed on the 
teaching pendant. 

[0066] Then, a target workpiece is placed at a prede- 
termined position in the field of the vision of a camera, 
and the focalization, stop, and zoom of a lens are ad- 45 
justed (step S12). At this time, the lens adjustment ring 
is operated to obtain an optimum state while watching 
the raw image displayed on the monitor screen of the 
teaching pendant. The adjustment ring is fixed in the op- 
timum state so that the adjusted state of the lens does so 
not change. This lens adjusting operation is easily per- 
formed with the configuration of the apparatus while 
watching the image near the target workpiece or the 
camera. 

[0067] Then, information as to where in a robot coor- ss 
diriate system the field of the vision of a camera corre- 
sponds to (step S1 3) is set. This setting is referred to as 
camera calibration, wherein parameters for use in con- 



verting the position (in terms of number of pixels) of the 
target workpiece detected in the image into the position 
(in terms of millimeters) in the robot coordinate system 
are set. The camera calibration can be embodied in var- 
ious methods, but in this example, the simplest.system, 
that is, a two-point calibration system will be described 
below. 

[0068] This system can be realized by setting two 
points in the field of the vision of a camera, and setting 
in the image processing apparatus the distance (mm) 
between the two points, the positions (pixels) of the two 
points in the image, and the position (X, Y) (mm) in the 
robot coordinate system, values can be set and inputted 
on the image processing apparatus manipulating menu. 
To set the position in the image, an image containing 
two points is fetched to the image memory, and the po- 
sition is specified on the displayed image. 
[0069] Then, an application program is generated 
such that the image processing apparatus can recog- 
nize the position of a target workpiece, and various pa- 
rameters of an image analyzing method are set (step 

514) . The application program is generated by the op- 
eration of the image processing apparatus operation 
menu of the teaching pendant. There are various image 
analyzing methods. In this example, a template match- 
ing method is used to set a template image of a target 
workpiece as a parameter. When the image is set, a tar- 
get workpiece is specified by being encompassed by a & 
rectangular window in the displayed image. The speci- ? 
tied template image is stored in the data memory togeth- s 
er with a detection threshold, an application program, 
etc. 

[0070] If the image analyzing method is set, the target 
workpiece is moved within the field of the vision to con- 
firm whether or not it can be correctly detected (step 

515) . When the target workpiece is moved to various 
positions in the field of the vision, a raw image from the 
camera is displayed on the teaching pendant. When a 
detection test is selected on the image processing ap- 
paratus manipulating menu, an image fetching unit 
stores an image from the camera in the image memory. 
Then, the image analyzing unit carries out matching 
check with the template image which has already been 
set, and detects the target workpiece. The detected po- 
sition is displayed in the graphics of the image. 
[0071] If it is confirmed that the target workpiece can 
be correctly detected, then a target workpiece can be 
placed at an appropriate reference position, and the po- 
sition is set as a reference position in the image process- 
ing apparatus (step S16). When the target workpiece is 
set, the raw image from the camera is displayed to sup- 
port the target workpiece to be easily centered in the 
field of the vision. To set a position as a reference posi- 
tion, a reference position setting item is selected from 
the image processing apparatus manipulating menu. 
Then, as in the case of the above described test, an im- 
age is fetched and processed, and the target workpiece 
is detected. The detected position in the image is stored 
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in the data memory as a value in the robot coordinate 
system using the camera calibration data which has al- 
ready been set. 

[0072] Furthermore, a robot handling operation to be 
performed on the target workpiece located at the refer- 
ence position is indicated and set (step S17). When an 
operation is switched from setting ot the image process- 
ing apparatus to setting ot the robot, it is not necessary 
to switch operating units, but is necessary only to switch 
operation menus. The robot is manually operated by us- 
ing an exclusive hard key ot the teaching pendant, and 
indicates the handling operation. 
[0073] Before starting the handling operation, a robot 
program is prepared such that an instruction to detect a 
target workpiece is issued to the image processing ap- 
paratus to wait for the completion of processing an im- 
age. A position adjustment instruction is added to the 
handling point and the points before and after the han- 
dling point so that the operation can be adjusted from 
the image processing apparatus. 
[0074] Described above is the outline of the manipu- 
lations to be performed when an application is set up. 
The operation to be performed when the setup applica- 
tion is executed will be described below by referring to 
the flowchart shown in Fig. 14. 

[0075] First, when a robot program is activated (step 
S21), an image process instruction in the robot program 
is executed, and the image processing apparatus is no- 
tified of the execution through the communications in- 
terface (step S22). The image processing apparatus 
fetches an image from the camera, and stores it in the 
image memory (step S23). 

[0076] Then, an image analysis unit performs tem- 
plate matching, detects a target workpiece, and displays 
it on the monitor screen (step S24). The position of the 
detected target workpiece on the image is converted in- 
to the position in the robot coordinate system, and the 
amount of the shift of the position from the predeter- 
mined reference position of the target workpiece, which 
has been set in advance, is computed (step S25). The 
computed amount of the shift is transmitted to the robot 
through the communications interface, and the robot is 
notified of the completion of the image processing (step 
S26). 

[0077] In addition, the detection result is drawn as 
graphics on the image information of the teaching pen- 
dant, and on the image information of an independently 
connected TV monitor. The maintainors of a production 
line can confirm at any time that the image processing 
apparatus is correctly working by watching the graphics. 
[0078] When the robot receives the image process 
completion notification, it adjusts and controls the posi- 
tion using the transmitted data of the amount of the shift 
to handle the target workpiece (step S27). 
[0079] As described above, since the image process- 
ing apparatus manipulating menu and the image infor- 
mation can be displayed on the portable teaching pen- 
dant for performing an manipulation relevant to an im- 



age processing according to the present invention, an 
operator can go close to a target workpiece or a camera, 
carrying the portable teaching pendant, and, performs 
an operation such as changing the position of the target 

5 workpiece as necessary, watching the image informa- 
tion displayed on the teaching pendant, or can perform 
manipulation of the image processing apparatus. 
[0080] Furthermore, if the image processing appara- 
tus can be built in the robot controller, and an image 

10 processing manipulation function and an image moni- 
toring function are incorporated into a portable robot 
teaching pendant, then the operability of the system 
where a robot and an image processing apparatus are 
incorporated into one system can be further improved. 

is 

Claims 

1 . An image processing apparatus for a robot which is 
20 built in a robot controller, and has a portable teach- 
ing pendant connected thereto, comprising: 



a unit for fetching an image from a camera; 
memory storing image data from the camera or 
intermediate image data obtained in a stage of 
image processing; and 

a unit for converting image data from the cam- 
era, the image data from the camera stored in 
the memory, or the intermediate image data in- 
to a gray scale or a color scale, wherein: 
said teaching pendant comprises a display unit 
and a unit for manipulating image processing; 
and 

said display unit displays an image and indica- 
tion for manipulation of image processing, si- 
multaneously, or in a switching mode or super- 
position mode. 

An image processing apparatus for a robot which is 
designed independent of the robot controller, and 
has a portable teaching pendant connected thereto, 
comprising: 



25 



30 



35 



40 



a unit for fetching an image from a camera; 
45 memory storing image data from the camera or 

intermediate image data obtained in a stage of 
image processing; and 

a unit for converting image data from the cam- 
era, the image data from the camera stored in 
so the memory, or the intermediate image data in- 

to a gray scale or a color scale s wherein: 
said teaching pendant comprises a display unit 
and a unit for manipulating image processing; 
and 

55 said display unit displays an image and indica- 

tion for manipulation of image processing, si- 
multaneously, or in a switching mode or super- 
position mode. 
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3. The apparatus according to claim 1 or 2, further 
comprising: 

a unit for displaying and superposing geomet- 
ric graphics on the image displayed on the display 
unit in accordance with the operation procedure of s 
image processing and specifying an image 
processing with respect to the image. 

4. An image processing apparatus for a robot which is 
built in a robot controller, comprising: io 

a unit for fetching an image from a camera; 
memory which stores image data from the cam- 
era or intermediate image data obtained in a 
stage of image processing; and is 
a unit for converting image data from the cam- 
era, the image data from the camera stored in 
the memory, or intermediate image data into a 
gray scale or a color scale, wherein: 
a portable teaching pendant is connected to 20 
said robot controller through a cable; and 
said teaching pendant comprises a unit for gen- 
erating or editing a robot program, a unit for op- 
erating the robot, and a display unit, and can 
display on the display unit an image converted 25 
into the gray scale, and comprises a un it for ma- 
nipulating image processing; and 
said display unit displays, indication for gener- 
ating or editing of the robot a program and in- 
dication for manipulation of image processing, 30 
together with an image, simultaneously, or in a 
switching mode or superposition mode. 

5. The apparatus according to claim 4, further com- 
prising 3S 

a unit for displaying and superposing geomet- 
ric graphics on the image displayed on the display 
unit in accordance with the operation procedure of 
the image processing and specifying an image 
processing with respect to the image. 40 

6. The apparatus according to claim 1 , 2, or 4, wherein 
a part of the operation unit of the teaching pendant 
is configured by a touch panel. 

45 

7. The apparatus according to claim 4, further com- 
prising a unit for incorporating an instruction to proc- 
ess an image into a program of a robot. 

so 
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